Neural networks represent an alterative approach to information processing than the serial algorithmic computers of the type envisioned by John von Neumann. In a computer of the von Neumann type a single processing device executes an explicit program in a step-by-step serial fashion. Many information processing problems can be solved by this approach, as witnessed by the recent explosive rise in the number of computers and in the number of applications susceptible to computer processing.
There are limitations to type of information processing which can be performed by a computer based on the von Neumann model. Before such a computer can be employed to perform an information processing function, a person must understand the nature of the problem to be solved and produce an explicit program of instructions to solve this problem according to an algorithm. An algorithm is a mechanistic or recursive procedure for the solution of a problem. The existence of such an algorithm is a necessary condition for information processing employing a computer based on the von Neumann model. In addition, it is further necessary that a person translate that alogorithm into the explicit program required by the particular computer.
There are numerous important information processing tasks which are difficult or impossible to perform using von Neumann type computers. In some cases algorithms have not yet been developed for these information processing tasks. In other cases it may be impossible to develop an algorithm of the type required by von Neumann type computers. Information processing tasks of this type include pattern recognition, understanding continuous speech, identifying handwritten characters, and identifying objects having varying orientations.
Problems of this type are of the sort that people can do well while current computers do poorly or not at all. Neural networks are attempts to exploit the supposed organization of the human brain to attack these problems. Neural networks are collections of many individual computing elements called neurons. These neurons are coupled together in complex networks in which any single neuron receives input from many other neurons. The coupling from any neuron to another neuron many be strong or weak.
There are two methods commonly employed in selection of these internal representation of the network. In the most general of neural networks a feedback learning process is employed to change the strength of connection between neurons and to change the transfer function of each neuron to accommodate the particular information processing task at hand. Usually this is done by presenting inputs to the neural network and the corresponding desired outputs, and allowing the network to determine the proper internal representation for generating the desired output from the input. An important class of neural networks, including those constructed according to the Hopfield model, do not yet incorporate this dynamic learning process but require preselected connection strengths. This permits the construction of large networks by simplifying the construction of individual neuron circuits and their connections. However, this advantage in construction is offset by the requirement that the connection strengths corresponding to a set of desired stored vectors be determined.
Hopfield has proposed a neural circuit model in, e.g. "Simple Neural Optimization Networks: an A/D converter, a signal decision circuit, and a linear programming circuit", by J. J. Hopfield and D. W. Tank, IEEE Transactions on Circuits and Systems, vol. CAS-33, No. 5, May 1986, pages 533-541. Neurons are modeled as processing input/output devices. The synaptic weight T.sub.ij describes the extent to which the input of the i-th neuron is driven by the output of the j-th neuron. A positive value of T.sub.ij indicates that the j-th neuron excites the i-th neuron. A negative value of T.sub.ij indicates that the j-th neuron inhibits the i-th neuron. This proposed system implements the neurons as operational amplifiers connected by linear resistors or conductances. The operational amplifiers realize the input/output relation. When T.sub.ij is greater than 0, a normal operational amplifier is employed. When T.sub.ij is less than 0, then an inverting operational amplifier is used. The linear resistors or conductances realize the absolute values of the synaptic weights T.sub.ij. The Hopfield model generally does not include self-feedback for each neuron circuit in which a portion of the output of each neuron circuit is feedback to its input. Such a network will settle to one of a limited number of steady state equilibria for each given set of inputs, synaptic weights and self-feedback weights (if employed).
A practical neural network needs to be programmable. That is, the synaptic weights T.sub.ij and the self-feedback weights should be alterable by the user. This enables the neural network to be applicable to differing problems. Changing the synaptic weights and the self-feedback weights alters the location and number of the steady state equilibria of the network. The Hopfield model achieves this alterability using variable linear resistances for the synaptic weights.
Typical neural networks include many individual neuron circuits. Practical uses of neural networks often require hundreds or thousands of neuron circuits. If information processing using neural networks is to achieve wide use, the construction of neural networks must be adapted to permit large number of neuron circuits to be employed. This implies an accommodation between the model of individual neuron circuits and their connectivity and circuits that are readily and abundantly realizable in hardware. Due to the large size of such neural networks, it would be particularly advantageous to construct neural networks in an integrated circuit technology which is relatively inexpensive, such as analog metal oxide semiconductor very large scale integration (analog MOS VLSI).
There are some obstacles in VLSI implementation of neural networks including: 1) the necessary dense connectivity between neuron circuits; 2) the requirement that the linear resistive element from output j to input i (T.sub.ji) be equal to the linear resistive element from output i to input j (T.sub.ij); and 3) the problem of implementing variable linear resistive elements in silicon. The first problem is not different from existing challenges in VLSI designs and can be dealt with using standard and evolving techniques. It is known that the equality of the symmetric linear resistive elements is a necessary condition to ensure convergence of all solutions to equilibria only. It is also known that small imperfections generally preserve the convergence to equilibria but may introduce dynamic (non-equilibria) steady states such as oscillations. Thus this requirement is a problem that may be possible to endure. The last obstacle, that of implementing variable linear resistance elements in silicon, appears to be the major obstacle to constructing the existing continuous-time models of artificial neural networks in analog VLSI; in particular, dynamic feedback neural networks of the Hopfield type.
There have been several proposed approaches to the problem of implementing neural networks in MOS VLSI. Firstly, the programmability of the network can be traded for simplicity of implementation. This can be achieved using fixed resistive elements to provide the weights between the neurons as proposed in "VLSI Implementation of a Neural Network Memory with Several Hundreds of Neurons", by H. P. Graf, L. D. Jackel, R. E. Howard, B. Straughn, J. S. Denker, W. Hubbard, D. M. Tennant and D. Schwartz, AIP Neural Networks for Computing, pages 182 to 187, 1986. Secondly, the neural weight can be restricted to +1, -1 or 0 and implemented in pass transistors and switches. This was proposed in "A Novel Associative Memory Implemented Using Collective Computation", by M. Sivilotti, M. Emerling and C. Mead, 1985 Chapel Hill Conference on VLSI, pages 329 to 342, 1985. Other proposed designs modify the weights but do not retain the qualitative dynamic properties from an analytic point of view.